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Effectiveness in distance education has been scrutinized for years, but only recently has literature focused on 
Web-based learning. As more online degree programs are established, the need for evaluation methodologies 
and processes guiding those activities increases. This paper describes on-going research to develop an 
evaluation plan for a fully online graduate degree program in HRD, including the theoretical framework, 
process, methodology, and resulting evaluation plan. Future work includes plan implementation and 
subsequent reporting. 
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Although the quality of distance education has been discussed for over a decade, it is only in the past few years that 
the literature has focused on Web-based or e-leaming (Calvert, 2003). Online courses and programs are continuing 
to gain popularity among both institutions of higher education and business and industry as indicated by U.S. News 
who currently lists 263 accredited online graduate degree programs on their website (U.S. News and World Report, 
US News Online, 2004). 

This paper describes the on-going research undertaken to develop an evaluation plan for a fully online graduate 
degree program in HRD. Through a review of the literature and consultations with program administration and 
staff, a plan was developed with the purpose of assessing the state of the program in reference to its goals, 
determining a roadmap for program improvement, and providing a framework for future program decision-making. 
The theoretical framework, process, methodology, and resulting plan for evaluating the online degree program are 
all described herein. Specifically addressed is how an online HRD master’s degree program evaluation was 
developed and translated into a plan for use in any online program evaluation in HRD. Future work on this project 
includes implementation of the evaluation plan and subsequent reporting of the results. 

Problem Statement 

As more and more completely online degree programs are established, the need to develop evaluation 
methodologies and processes to guide that assessment practice increases. While much attention has been given to 
the quality of online courses or components of courses, much less focus has been given to the evaluation of online 
degree program as a whole. Past research has compared online learning to face-to-face learning (Hoben, Neu & 
Castle, 2002), explored the effectiveness of online tools such as discussion boards and chat rooms (Spatariu, 
Hartley, & Bendixen, 2004), assessed aspects of courses (Roblyer & Ekhaml, 2000), addressed evaluating effective 
online instruction (Graham, Cagiltay, Lim, Craner & Duffy, 2001; Wending & Johnson, 1999), and assessed the 
value of online courses in specific fields of study (Carmichael, 2001; McMaster, 2002). There have also been many 
articles concerning the success or failure of a variety of technologies used in this environment (Feldman, 2002; 
Smith, 1998). Most of the literature about online evaluation of teaching and learning has been aimed at the 
individual course level. While these studies make significant contributions to the best practice and theoretical 
framework for the use of individual tools and processes, they do not specifically explore program evaluation in a 
holistic manner. 

Background of the Project 

In the process of undergoing a mandated graduate program review, two observations surfaced. The first observation 
was that the formal review process was focused on outcomes and not necessarily on improvement. The second 
observation was that while all programs have similarities, there are certain characteristics of online degree programs 
that are not addressed in standardized review processes. The degree program under review is a 36 credit-hour 
master’s degree program in HRD. The program was established through grant funding in 2001 and the first fully- 
online degree students entered the program in the fall of 2002. The program started with a cohort-based model and 
moved to an open-enrollment model at the beginning of its third year. Courses were developed in a systematic 
fashion, specifically for the online format. WebCT has been used as the primary learning management system since 
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its inception and students are required to participate in a short on-campus orientation prior to enrolling in their first 
course. The entire program is delivered online. The on-campus program from which this program evolved is no 
longer accepting new students, so this is the only way that students can enroll in this program. The faculty teaching 
in the program has changed over the past two years with only two of the original five insttuctors still employed at 
the institution. While the faculty turnover has been an issue, the program administration has stayed relatively stable 
and has been a stabilizing force. 

Evaluation 

Evaluation theory has its roots in social inquiry and the desire for accountability and control. Depending upon the 
goals of the evaluation, different methodologies and strategies are used to guide inquiry. The methodologies and 
strategies are selected based on the type of evaluation required. Evaluation falls into three main types, that oriented 
toward the construction of knowledge, those orientated toward placing value, and those oriented toward how 
information is used (Alkin & Christie, 2004). 

Evaluation is essential to maintaining and improving a degree program. Evaluation can be further broken into 
two distinct categories, formative and summative (Scriven, 1967). Formative evaluation focuses on processes and 
summative evaluation focuses on final products (Van der Veer Martens , n.d.). The purpose of this evaluation is to 
sttengthen and improve the program. Therefore, a formative evaluation was selected to guide this inquiry. 

Formative evaluation can come in a variety of forms. The purpose of needs assessment is to gather information 
on whether or not there is a substantive need for an action, evaluability assessment involves deciding whether or not 
the program or entity can be feasibly evaluated, structured conceptualization defines the program or technology, the 
target audience, and the possible outcomes, implementation evaluation looks at program delivery, and process 
evaluation is designed to review implementation of the program, to provide advice in a formative manner to improve 
implementation and to track systems change (Rossi, 2004; Alkin & Christie, 2004) 

Program Evaluation 

Evaluators have been criticized in the past for focusing on outcome evaluation and excluding the process side, or 
focusing on process evaluation without examining outcomes (Fleischman & Williams, 1996). The framework 
presented here incorporates both the process and outcome sides in an effort to not only determine the effects of the 
program, but also understand how the program might be improved. 

This evaluation must include both process and outcome components. The process evaluation component focuses 
on describing the program implementation including students, faculty and services provided. The outcome 
evaluation component focuses on assessments of program effectiveness with respect to measurement standards in 
the community of higher education. 

Graduate Degree Program Evaluation 

In accredited institutions of higher education, graduate programs are required to undergo formal program assessment 
at regular intervals. This assessment is standardized and usually occurs every five to eight years. The formal review 
in this case is a very sttuctured process, one that is relatively the same for all programs, making no distinct 
considerations for the type of program or its characteristics. While this process reveals the success or failure of 
programs to meet predetermined goals, it does not focus on program improvement. 

Evaluation in the Online Environment 

In 2000, the Institute for Higher Education Policy, in association with Blackboard and the National Education 
Association, prepared a report titled Quality on the line: Benchmarks for success in Internet-based distance 
education. This study involved a thorough review of the literature in search of what constitutes quality in online 
education. The researchers used the case study method to compile their information. First, a literature review was 
conducted to compile benchmarks already reported by other researchers and organizations and in reports and other 
publications. The literature review resulted in 45 benchmarks. Next, they identified institutions that were already 
leaders in Internet-based education, and finally they visited each of the selected institutions and assessed the degree 
to which they incorporated the benchmarks. As a result of the study a final list of 24 benchmarks were established 
that are considered most essential to the success of an Internet-based distance education program at any institution. 
The 24 benchmarks are outlined in Table 1. 
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Table 1. Benchmarks essential for program success. 



Institutional 

Support 

Benchmarks 


A documented technology plan that includes electronic security measures is in place and operational to 
ensure both quality standards and the integrity and validity of information. 


The reliability of the technology delivery system is as failsafe as possible. 


A centralized system provides support for building and maintaining the distance education infrastructure. 


Course 

Development 

Benchmarks 


Guidelines regarding minimum standards are used for course development, design, and delivery, while 
learning outcomes determine the technology being used to deliver course content. 


Instructional materials are reviewed periodically to ensure they meet program standards. 


Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as 
part of their course and program requirements. 


Teaching/ 

Learning 

Benchmarks 


Student interaction with faculty and other students is an essential characteristic and is facilitated through 
a variety of ways, including voice-mail and/or e-mail. 


Feedback to student assignments and questions is constructive and provided in a timely manner. 


Students are instructed in the proper methods of effective research, including assessment of the validity 
of resources. 


Course 

Structure 

Benchmarks 


Before starting an online program, students are advised about the program to determine (1) if they 
possess the self-motivation and commitment to learn at a distance and (2) if they have access to the 
minimal technology required by the course design. 


Students are provided with supplemental course information that outlines course objectives, concepts, 
and ideas, and learning outcomes for each course are in a clearly written, straightforward statement. 


Students have access to sufficient library resources that may include a “virtual library” accessible through 
the World Wide Web. 


Faculty and students agree upon expectations regarding times for student assignment completion and 
faculty response. 


Student 

Support 

Benchmarks 


Students receive information about programs, including admission requirements, tuition and fees, books 
and supplies, technical and proctoring requirements, and student support services. 


Students are provided with hands-on training and information to aid them in securing material through 
electronic databases, interlibrary loans, government archives, news services, and other sources. 


Throughout the duration of the program, students have access to technical assistance, including detailed 
instructions regarding the electronic media used, practice sessions prior to the beginning of the course, 
and convenient access to technical support staff. 


Questions directed to student service personnel are answered accurately and quickly, with a structured 
system in place to address student complaints. 


Faculty 

Support 

Benchmarks 


Technical assistance in course development is available to faculty, who are encouraged to use it. 


Faculty members are assisted in the transition from classroom teaching to online instruction and are 
assessed during the process. 


Instructor training and assistance, including peer mentoring, continues through the progression of the 
online course. 


Faculty members are provided with written resources to deal with issues arising from student use of 
electronically-accessed data. 


Evaluation 

and 

Assessment 

Benchmarks 


The program's educational effectiveness and teaching/learning process is assessed through an evaluation 
process that uses several methods and applies specific standards. 


Data on enrollment, costs, and successful/innovative uses of technology are used to evaluate program 
effectiveness. 


Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness 



Note. From “Quality on the line: Benchmarks for success in Internet-based distance education" by The Institute for 
Higher Education Policy, 2000. 



In 2001, a formative evaluation of e-learning was completed for a Canadian military organization (Broadbent & 
Cotter, 2003). The evaluation focused on a 6-week web-based course preparing students for instruction on military 
management skills. The evaluators used Stufflebeam’s CIPP model (context, input, process, product) as a 
framework for their evaluation plan. The plan employed four data collection methods: written questionnaires from 
students and instructors before and after training, focus groups with students and instructors before, during, and at 
the end of the course, direct observation during the training (monitoring web discussions, observations during 
classroom sessions, and document review.) Broadbent and Cotter concluded that although evaluation is 
fundamental to grasping the value of an e-learning program that the plans will vary depending on the goals of the 
program and the budget assigned to the evaluation. 
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Assessing Quality in the Selection of a Degree Program 

When deciding on whether or not to select an online degree program, students face an enormous number of 
considerations. To aid in the selection process, many online websites devote space to pointing student in the right 
direction. One site lists four considerations when assessing the quality of an online master’s degree program: Does 
the program use professor-guided web-threaded discussion groups?, (insisting that the omission of this leads the 
student toward correspondence courses). What does the graduate program use for a course hosting platform? 
(suggesting student-friendly platforms such as eCollege, WebCT or Blackboard), Does the course use current videos 
and lecture materials? (Telling students to beware of “canned” courses), and What is the status of the school's 
accreditation? (Florida Atlantic University, 2003). 

Another site tells students to ask the following questions: How is the course presented? (find a school that 
incorporates innovation). How do students interact with each other? (find a program that requires interaction). Are 
the insUuctors qualified? (looks for staff and support personal dedicated to their online programs). What kind of 
reputation does the school have? (look for a strong program, not just school). How are students evaluated? (ensure 
the programs involved apply what has been learned). What kinds of library facilities are available? (look for a 
system that ensures that reference materials are available from anywhere) (Obringer, n.d.). 

Theoretical Framework 

The model guiding this evaluation is the CIPP model originally developed by Donald Stufflebeam in 1971 as part of 
the Phi Kappa Delta Commission on Evaluation. The CIPP acronym was formalized in 1983 and updated most 
recently in 2002 (Stufflebeam, 1974; 1983; 2002). The CIPP acronym stands for the core concepts of the model, 
context, inputs processes and products. This model recognizes essential types of decisions encountered in education, 
planning, programming, implementing, and recycling (Wentling, 1980). Context evaluation reflects the 
environment, identifies needs, and forms goals and objectives. Input evaluation assesses the competing ways to 
achieve the goals specified in the context evaluation. Process evaluation reviews how the program operates. 
Product evaluation focuses on program results, connecting outcomes with the other measurements taken in the 
earlier areas of evaluation. Since the model’s inception, Stufflebeam has added the additional concepts of 
effectiveness, sustainability, and transportability. This model was chosen for two main reasons, 1) the model places 
emphasis on guiding planning, programming, and implementation efforts and 2) the model emphasizes that the most 
important purpose for evaluation is improvement (Stufflebeam, 2002). 

Research Design and Development of the Evaluation Plan 

In order to result in a useful and effective evaluation, stakeholders would need to be identified. In this case, the 
primary stakeholders were identified as the program faculty. The faculty asked for the evaluation and it is they who 
will be primarily affected by and interested in its findings. Secondary stakeholders included the department head, 
the director of graduate programs, and the students. All of these entities have an interest in the evaluation results but 
will not be directly tasked with any of the improvement efforts that may result. 

An essential component to any evaluation plan is to obtain agreement from the stakeholders as the goals and 
expectations of the evaluation process (Stufflebeam, 2002). The first step in this evaluation plan was to meet with 
the stakeholders, in this case, the faculty of the Training and Development program, and agree on the evaluation 
parameters and responsibilities. A formal program review was simultaneously occurring through the Graduate 
School, so there were already some program goals in place, but those goals were largely mandated by governmental 
and legal entities and were focused on outcomes only. Since this evaluation was to focus on improvement, other 
goals, ones more highly regarded by the stakeholders, would have to be developed. As a result, the primary 
stakeholders were interviewed to determine their vision of the program goals and evaluation needs. The results list 
was reviewed and agreed upon by all primary stakeholders. These goals included: 

• Maintain a program focus and course offerings that compare favorably with other top ranked HRD 
programs. 

• Incorporate the benchmarks determined to promote quality in online programs. 

• Offer a set of courses that provide students with the proper competencies based on current needs and trends 
in the field of HRD. 

In following the CIPP model, the next part of the evaluation will be the context analysis. This analysis will 
determine the environment, needs, assets, and problems in the program. Although the environmental questions could 
be answered by review of extant data, it was determined that there were several groups that would have input into 
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the needs and problems of the program, the students and the instructors. Students and instructors will both have 
varying needs and perspectives on the program. They are also two easily accessible groups. 

In the input analysis, the evaluation must look at the competition and the strategies and use these as a 
comparison to those of the program. For this analysis there will be two components, a review of the literature to 
review published information about the competition and a Web search for information about the competition. The 
focus of these searches will be to investigate other programs in search of one or more that could serve as models and 
to compare our program strategies with those of other successful, established programs. 

For the process part of the evaluation, program activities must be monitored, documented and assessed. 
Although there had been no prior formalized program evaluation, there have been other evaluation activities 
occurring throughout the program’s existence. Each course that was taught was assessed through a standardized 
student end-of-course evaluation. There was also a large assortment of program documents that had been saved and 
stored. In addition, all of the program faculty and staff that had taught in the program will still be accessible to the 
evaluator. This will allow the evaluator to not only develop a current accurate profile of the program, but also delve 
into any situation where more information was needed. 

The product/impact part of the evaluation assesses the program’s reach to the target audience. This is the part of 
the evaluation concerned most with the students. Perspectives are at issue here as are any hard evidence of impact 
upon the intended target audience. Most likely, much of the evidence of impact will come from perceptions. This is 
one reason that multiple perspectives must be used to assess this aspect of the program. 

Effectiveness evaluation looks at the quality and significance of the program outcomes. Effectiveness conjures 
up many of the standard outcome measures such as graduation rate and retention. Since this program is relatively 
new and the first cohort has yet to graduate, some of the standard quantitative measures will not yield useful 
information. So, in addition to measures of retention and grade point average, this part of the evaluation will elicit 
effectiveness information from the current beneficiaries of the program, the students. In future evaluations, it will be 
important to also elicit information from other beneficiaries, such as area employers and other community 
stakeholders. 

The sustainability aspect of this evaluation must look at the extent to which the program's contributions are 
institutionalized over time. Here the evaluation should find out which program attributes should be maintained and 
which should be redirected and/or removed. The program beneficiaries (the students), the program faculty, and the 
program administration will need to have input into this part of the evaluation. In addition, the program budget 
should be analyzed for sustainability and effectiveness for its intended goals. 

Finally, the transportability aspect of the evaluation will look at the extent to which the program could and/or 
should be adapted or implemented elsewhere. As online learning continues in popularity among institutions of 
higher education, there will be a growing need to identify those program and practices that are successful and 
effective. To move online learning forward, the structures and processes from the more successful program will 
need to be adapted by other institutions and programs. In this case, there are already initiatives under foot to 
establish additional online programs. This part of the evaluation will assist new programs in establishing similar 
programs in their own environments. 

The Evaluation Plan and Methodology 

Each of the concepts of the CIPP evaluation model was used as a lens with which to determine the aspects of the 
program that should be addressed and sources from which data should be collected. This resulted in a list of 18 
aspects of the program that should be investigated. This list of aspects and sources of information is displayed in 
Table 2, along with concept of CIPP model that advised its inclusion. 

Once the components of the evaluation were determined, the evaluation plan could be developed. Many 
considerations were explored when deciding on how the data would be collected and via what instruments. The 
review of the existing data such as the course materials, program documents, and standard program measures are 
easily accessible and readily available. The collection and review of this data and other competitive information is 
relatively straightforward. Several of the evaluation components required input from program faculty. In order to 
obtain a well-rounded perspective and the fact that all faculty members are relatively accessible, a focus group 
format was chosen. 

There are only three program administrators so an interview methodology was selected for this group. This will 
not only yield rich data, but was also selected because of the ability to use probing techniques for this group that 
holds so much of the operating and process information. Input is also required from current students enrolled in the 
program. Because this is a distance program, interviewing and group techniques are not practical. As a result, 
student input will be elicited via an online survey. The final part of the plan is a review of the evaluation that looks 
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at what aspects of the program can and should be replicated and what aspects should be altered or discontinued. A 
summary of the elements of the evaluation plan is found in Table 3. 



Table 2. Components of the evaluation plan. 



Evaluation Element 


CIPP Concept 


A review of program documentation for information on problems, needs, assets, 
and environment of the program 


Context 


Input from program administrators and faculty for information on problems, needs, 
assets, and environment of the program. 


Context 


Input from program students to gather information on problems, needs, assets, and 
environment of the program 


Context 


A review of other HRD programs to determine the top-tiered programs in HRD 


Input 


A comparison of program’s focus and course offerings with the identified top-tiered 
programs 


Input 


A review of literature in HRD to determine current trends and issues on which 
HRD master’s programs should be focused 


Input 


Review of existing program end-of-course evaluations 


Process 


Input from program staff and administration to gather information on program 
activities and processes. 


Process 


Input from current students to assess perceptions of program impact 


Impact 


Interview with faculty to assess perceptions of program impact 


Impact 


Review of student GPAs 


Effectiveness 


Review of retention figures and progress toward degree completion. 


Effectiveness 


Input from current students as to program effectiveness 


Effectiveness 


Input from program faculty and administration about program aspects that should 
be maintained, changed or removed 


Sustainability 


Input from current students about program aspects that should be maintained, 
changed, or removed. 


Sustainability 


Analysis of budget and spending 


Sustainability 


Review of the evaluation to determine what aspects of the program can be used in 
establishing and/or improving other online degree programs. 


Transportability 



Table 3. 



Resulting elements of the evaluation plan. 



Evaluation Element 


CIPP Concept Addressed 






A review of existing program documentation including 
end-of-course evaluations, student GPAs, retention 
figures, progress toward degree completion, budget 


context, process, effectiveness, sustainability 


Review of the literature and top-tiered programs in HRD 
and comparison with program under evaluation in regard 
to course offerings and focus 


input 


Focus group with program faculty focused on eliciting 
information about problems, needs, assets, environment, 
and impact of the online program 


context, impact 


Interviews with program administrators about problems, 
needs, assets, environment, program activities and 
processes, aspects of the program that should be 
maintained, changes and removed 


context, process, sustainability 


Survey of current students about problems, needs, assets, 
environment, perception of program impact and 
effectiveness, and aspects of the program that should be 
maintained, changed, and removed 


context, impact, effectiveness, sustainability 


Review of the evaluation for best practices and 
suggestions for future online program implementation 


transportability 
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Next Steps 

Evaluation plans should be working documents with the ability to change as new information informs the 
evaluation. This evaluation plan is no different and will be altered as needed. With program stakeholder approval 
of the evaluation plan completed, the instrument development will be the next step in the process. This will be 
undertaken with a continued systematic approach. The data collection instruments will be developed based on the 
IHEP benchmarks and other measures of quality in the literature (Institute for Higher Education Policy, 2000; 
Scanlan, 2003). The student survey instrument will be assessed for face validity by program faculty familiar with 
evaluation processes. It will then be pilot-tested with a group of students outside the sample and assessed for 
reliability. Focus group and interview questions will go through similar assurance processes. Data collection 
procedures will be formalized and then data collected. This will be followed by data analysis, development of 
conclusions and reporting. Both quantitative and qualitative methods of analysis will be used to interpret the data. 
It is important to realize that all evaluations are limited by time and resources. The resulting evaluation plan is not 
meant to be the be-all, end-all way to evaluate online programs, but it will serve as an effective model for online 
program evaluation working within the constraints of the university system. 

Implications for HRD 

This lack of evaluation of completely online programs can be explained with a variety of reasons such as a lack of 
university-based fully online degree programs to evaluate in the past and a limited understanding about the 
benchmarks in assuring quality in those programs. This research attempts to close the gap in the literature by 
developing a theoretically sound, research-based approach to online program evaluation by reviewing the literature 
and other relevant sources to develop a practical, yet effective approach to developing a comprehensive evaluation 
plan for online degree programs. Over the next few years it is anticipated that the number of online degree programs 
will continue to expand. Like all markets, at some point supply will exceed demand. At that point, weaker programs 
will disappear. Effective program evaluation is the best way to achieve continuous program improvement. A 
review of the literature results in little evaluation research focused on on-line-only programs. This may be because 
these programs are relatively new and have not been in existence or stable long enough to collect meaningful 
information. The evaluation plan resulting from this project will help guide HRD program administrators in how to 
structure their own evaluations and inform them of the types of data that they may want to collect. The hope is that 
the processes and instruments developed will serve not only for ongoing assessment of this online program, but be 
applicable to the assessment of other online degree programs, locally and at other institutions. 
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